Storage servers, especially those configured for use in distributed storage systems, contain many storage devices. This makes it highly likely that there will be multiple storage device failures over the lifetime of a storage server. Because of this, storage servers are designed so that storage devices may be “hot-swapped,” or replaced without powering down the server.
Storage servers within distributed storage systems (i.e. storage systems comprising multiple storage servers that communicate in order to act as a single coherent service), could possibly have tens of storage devices, with the distributed storage system itself comprising thousands of storage servers. Because of this it becomes extremely problematic and laborious to easily identify and “hot-swap” a failed storage device in such an architecture having potentially millions of storage devices. First of all, it is difficult for a technician to identify a specific storage server that contains the failed storage device. Even if the technician manages to identify the storage server housing the failed storage device, there is no way he can immediately and readily identify the failed storage device within the server which houses many storage devices including the failed one.
Further, distributed storage systems are built to expect storage device failures, and ensure that data is not lost when a storage device fails. Existing distributed storage systems achieve this by storing any given piece of data multiple times, such that each of the copies/replicas are stored on different storage devices, and usually on different servers, so that the distributed storage system is not affected if a storage device or server fails. If a client tries to access data on a storage device that has failed, it is automatically redirected to retrieve the data from one of the replicas.
The exact number of replicas maintained by a distributed storage system is a policy decision and is a trade-off between how important the data is and how much extra cost will be incurred by having to buy more storage devices to hold the redundant copies. When a storage device fails or is decoupled from a server in such a distributed storage system, the system notices this and makes new replicas of the data that was contained in the failed storage device and/or the other storage devices that have also been decoupled from the server housing the storage device by copying the data from the remaining replicas stored on other servers in order to get back to the required level or redundancy. This process is called replication, and is costly in terms of both network traffic and server loading. However, when a storage device fails, if the failed device is not replicated, the level of redundancy and hence reliability of the distributed storage system is no longer maintained.
Referring back to the initial problem of replacing a failed storage device within the aforementioned distributed storage system, we are faced with the problem of replicating not only the failed storage device but also, replicating any working storage device that may be knowingly or unknowingly decoupled from the server in the process of identifying and replacing a failed storage device. This results in significant network and CPU utilization, which is unnecessary and usually leads to wasteful use of system resources, as the working storage devices will soon be coupled again to the storage server, and therefore need not have their data replicated.
It would be desirable to resolve these issues.